Deep Learning for Traffic Scene Understanding: A Review

PARYA DOLATYABI ${}^1$ , (Graduate Student Member, IEEE), JACOB REGAN ${}^1$ , (Graduate Student Member, IEEE), AND MAHDI KHODAYAR ${}^{◻}$ , (Member, IEEE)

Department of Computer Science, University of Tulsa, Tulsa, OK 74104, USA

Corresponding author: Parya Dolatyabi (pad7492@utulsa.edu)

This work was supported in part by U.S. Department of Transportation (USDOT) under Grant 693JJ32350030.

• ABSTRACT This review paper presents an in-depth analysis of deep learning (DL) models applied to traffic scene understanding, a key aspect of modern intelligent transportation systems. It examines fundamental techniques such as classification, object detection, and segmentation, and extends to more advanced applications like action recognition, object tracking, path prediction, scene generation and retrieval, anomaly detection, Image-to-Image Translation (I2IT), and person re-identification (Person Re-ID). The paper synthesizes insights from a broad range of studies, tracing the evolution from traditional image processing methods to sophisticated DL techniques, such as Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs). The review also explores three primary categories of domain adaptation (DA) methods: clustering-based, discrepancy-based, and adversarial-based, highlighting their significance in traffic scene understanding. The significance of Hyperparameter Optimization (HPO) is also discussed, emphasizing its critical role in enhancing model performance and efficiency, particularly in adapting DL models for practical, real-world use. Special focus is given to the integration of these models in real-world applications, including autonomous driving, traffic management, and pedestrian safety. The review also addresses key challenges in traffic scene understanding, such as occlusions, the dynamic nature of urban traffic, and environmental complexities like varying weather and lighting conditions. By critically analyzing current technologies, the paper identifies limitations in existing research and proposes areas for future exploration. It underscores the need for improved interpretability, real-time processing, and the integration of multi-modal data. This review serves as a valuable resource for researchers and practitioners aiming to apply or advance DL techniques in traffic scene understanding.

INDEX TERMS Deep learning, traffic scene understanding, discriminative models, generative models, domain adaptation, classification, object detection, segmentation.

I. INTRODUCTION

The rapid evolution of deep learning (DL), particularly in computer vision, has initiated a new era of intelligent transportation systems. Researchers have made significant strides in advancing autonomous vehicles, traffic management, and pedestrian safety by fusing deep neural architectures with the complexities of traffic scenes. However, despite these advancements, critical challenges persist in effectively translating theoretical breakthroughs into robust, real-world applications, such as handling the variability of traffic environments, ensuring real-time processing, and achieving high accuracy under diverse conditions. This review aims to thoroughly explore these challenges by offering an in-depth analysis of the complex interactions between deep neural networks (DNNs), computer vision, and traffic scene understanding.

Previous studies have made substantial contributions to this field. However, they also exhibit certain limitations. For example, [1] provided an extensive survey on deep learning-based object detection in traffic scenarios, covering over 100 papers and highlighting challenges such as real-time performance, image quality degradation, and object occlusion. Autonomous driving technologies were investigated in [2], with a focus on DL methods for perception, mapping, and sensor fusion. They also pointed out limitations in multi-sensor integration and prediction accuracy. The authors of [3] covered deep learning techniques for object detection, semantic segmentation, instance segmentation, and lane line segmentation in autonomous driving. They highlighted key challenges such as high computational cost, real-time performance limitations, and occlusion issues-particularly in instance segmentation, where region proposal-based methods often struggle with small or occluded objects. Advanced methods, such as Adaptive Feature (AF) pooling, were suggested to improve efficiency in these scenarios. The study in [4] reviews methods based on artificial intelligence (AI), including convolutional neural networks (CNNs) and reinforcement learning, for tasks such as driving scene perception, path planning, and motion control. It discusses challenges including handling occlusion, particularly during scene perception, where occluded objects often hinder accurate detection and recognition. Despite their valuable insights, these studies face several critical shortcomings:

The associate editor coordinating the review of this manuscript and approving it for publication was Turgay Celik ${}^1$ .

1. Current review papers largely focus on discriminative models, with limited coverage of generative models crucial for synthetic data generation in traffic scenarios. Additionally, they offer limited discussion on domain adaptation (DA) techniques, which are essential for transferring models across different environmental conditions to enhance robustness in traffic scene analysis [1], [2], [3].

2. Current surveys often lack a detailed discussion on hyperparameter optimization (HPO) [1], [2], [3], [4], which is crucial for fine-tuning DL models in complex traffic scenarios. This omission is significant, as HPO enhances model efficiency, reduces training time, and improves real-time deployment feasibility.

3. Many existing survey papers fail to provide a thorough comparison of different deep learning architectures, especially among discriminative, generative, and domain adaptation categories [1], [4]. This omission is important, as such comparisons are essential for understanding the unique strengths and weaknesses of each approach, which plays a key role in making informed decisions about model selection and application.

4. Existing literature lacks a comprehensive comparative discussion of various models, including their advantages, disadvantages, and potential areas for future research [1], [2], [3], [4]. Such an analysis is essential for understanding the trade-offs between different models and identifying opportunities for advancing state-of-the-art solutions in complex traffic environments.

5. Current reviews often cover challenges like safety and hardware but neglect emerging areas such as Explainable AI (XAI) and real-time processing [2], [3]. These areas are essential for advancing traffic scene understanding and ensuring AI systems are transparent and effective in dynamic environments.

6. Current studies are outdated, as they do not include a review of the most recent developments in the field, thereby missing the latest advancements shaping DL applications in traffic scene understanding [1], [2], [3], [4]. This is a significant limitation, as staying updated with the latest research is crucial for providing a comprehensive and forward-looking resource.

To address these gaps, our paper studies the core computer vision techniques of classification, object detection, and segmentation, while also extending its analysis to cover advanced topics including action recognition, object tracking, path prediction, anomaly detection, scene generation, and image enhancement. By synthesizing findings from a broad spectrum of studies, our paper provides a holistic overview of the evolution from traditional image processing methods to advanced DL models, including Convolutional Neural Networks (CNNs), Generative Adversarial Networks (GANs), and Domain Adaptation models. It emphasizes the integration of these models into real-world applications such as autonomous driving, traffic management, and pedestrian safety, while also addressing challenges including occlusions, dynamic urban traffic environments, and varying weather and lighting conditions. The contributions of our paper are:

1. We present a categorized exploration of discriminative, generative, and DA models, including detailed analyses of YOLO (You Only Look Once) variants, Vision Transformers (ViTs), graph-based models, and various DA techniques. Our approach not only covers these models comprehensively but also emphasizes their advantages, such as improved accuracy in traffic scenarios with discriminative models, enhanced training through generative models for synthetic data creation, and increased robustness in real-world conditions with DA techniques.

2. HPO strategies are discussed in our work, with specific sections dedicated to each category of DL architectures-discriminative, generative, and DA. This provides insights into optimizing these models for better performance, ensuring they are fine-tuned for optimal results in traffic scene understanding.

3. Our review provides comparisons of different DL architectures across discriminative, generative, and DA categories, focusing on application frameworks, variance in datasets, performance metrics, and overall results, providing clear guidance on selecting the most effective models for traffic scene understanding.

4. A comparative discussion of discriminative, generative, and DA models is provided in our work, highlighting their advantages, disadvantages, and potential directions for future research. This detailed analysis helps to clarify the strengths and limitations of each model, guiding future development and innovation in complex traffic scenarios.

5. We highlight emerging research trends such as XAI, augmenting vision backbones with ViT and GNN, real-time traffic scene processing, overcoming data limitations using synthetic data, and enhancing perception via multi-modality and data fusion. This structured perspective is crucial for advancing traffic scene understanding and providing a thorough, future-focused review.

6. We provide an up-to-date review of the latest papers and advancements, incorporating the most recent research insights. Keeping current with these developments is crucial for delivering a thorough and future-oriented resource.

These contributions make our paper a more thorough and forward-thinking resource, offering a critical assessment of current research, highlighting limitations, and presenting new perspectives on traffic scene understanding. By addressing the shortcomings of previous studies and offering a clear articulation of current challenges faced, this review aims to inspire future research and development efforts that drive innovation in deep learning for traffic scene understanding.

The rest of this paper is organized as follows. In Section II, we introduce the discriminative DL models, including CNNs, region-based CNN (R-CNN) variants, YOLO, ViT, DETR (Detection Transformer), graph-based models, and capsule networks (CapsNets). Section III focuses on generative machine learning (ML) models, encompassing GANs, conditional GANs (cGANs), and variational autoencoders (VAEs). In Section IV, we explore DA models within the categories of clustering-based, discrepancy-based, and adversarial-based approaches. A comparative discussion of these models is provided in Section V. HPO techniques are detailed within each category. Finally, future research areas and concluding remarks are presented in Sections VI and VII, respectively.

II. DATASETS

In this section, we focus on the most popular datasets identified in the papers reviewed in our study. These datasets are widely used across various tasks in traffic scene understanding, including object detection, segmentation, 3D tracking, classification, and domain adaptation. They have been selected based on their frequency of citation, versatility, and relevance to core applications. For less popular, niche, or highly customized datasets, readers are referred to the corresponding references cited in the respective works.

Table 1 summarizes 11 widely used datasets in traffic scene understanding, categorized based on their applications and characteristics. COCO 2017, VOC2007, and Cityscapes are benchmarks for object detection and segmentation, offering extensive annotations for diverse object categories and urban scenes. KITTI and nuScenes focus on 3D object detection and multi-object tracking, with KITTI emphasizing structured environments and nuScenes extending to radar data and more dynamic scenarios. GTSRB specializes in traffic sign recognition, providing a targeted dataset for autonomous driving systems. 1043-syn is a synthetic dataset optimized for traffic scene classification, particularly under controlled lighting and object variation scenarios. For person re-identification, DukeMTMC-ReID is a key benchmark, supporting identity matching tasks across multi-camera setups.

These datasets vary in scale and context, with real-world datasets like BDD, Mapillary, and Cityscapes capturing diverse weather and geographic conditions, while synthetic datasets like SYNTHIA and 1043-syn simulate controlled scenarios for domain adaptation and classification. Large-scale datasets such as COCO 2017 and BDD provide extensive data for deep learning, whereas smaller datasets like VOC2007 and KITTI offer high-quality annotations for specific tasks. This combination of real-world variability, geographic diversity, and synthetic precision allows researchers to address multifaceted challenges in traffic scene understanding, leveraging the strengths of each dataset for robust model development.

III. DISCRIMINATIVE DL MODELS

Discriminative DL models, often based on CNNs, are crucial for understanding complex traffic scenes. They excel at distinguishing objects and patterns, enabling tasks like object detection, classification, and segmentation. In traffic contexts, these models accurately identify vehicles, pedestrians, and road signs, enhancing real-time analysis in video feeds. By leveraging discriminative DL, systems improve road safety and efficiency, assisting autonomous navigation, traffic flow analysis, and pedestrian behavior prediction. This advancement supports intelligent transportation systems and enhances overall road safety.

In the following sections, we examine various discriminative DL models-R-CNNs, YOLO variants, and attention mechanisms-tailored for traffic scene understanding. These models address tasks like object detection, semantic segmentation, and action recognition, influencing intelligent transportation systems. We also discuss HPO for these architectures and compare performance metrics, providing a comprehensive overview.

A. CNN

A CNN [5] is a DL model designed for grid-like data (e.g., images), commonly used in traffic scene understanding to analyze camera images. CNNs automatically extract features like objects, signs, and road markings, supporting real-time processing and enhancing road safety for autonomous vehicles.

A basic CNN for image classification follows: convolution, pooling, fully connected (FC) layers, and output. Convolution applies filters to generate feature maps, which are pooled and flattened before passing through FC layers, with a softmax layer for class probabilities.

The core CNN operation is convolution:

where $I$ is the input image, $K$ is the kernel of size $m\times n$ ,and (x,y)are output coordinates.

TABLE 1. A summary of the most popular datasets identified in the papers reviewed in our study, categorized based on their characteristics and key features. Approximate numbers are used for dataset sizes to account for variations across versions, releases, or documentation. These datasets are widely adopted for diverse tasks such as object detection, segmentation, 3D tracking, classification, and domain adaptation, with applications spanning real-world scenarios and synthetic simulations. The inclusion of train and test sizes, along with geographic or virtual origins, highlights the diversity and specificity of these datasets in advancing traffic scene understanding. The term Varies in the Image Size column indicates datasets with images of multiple resolutions. Trainva/ refers to a combined set of training and validation images. Fine and Coarse denote levels of annotation granularity, with fine being pixel-accurate and coarse being approximate or less detailed.

Next, a ReLU activation function introduces non-linearity: $f_{\text{ReLU }}\left(x\right)=max\left(0,x\right)$ . A pooling layer reduces spatial dimensions to decrease parameters and computation. Max pooling is defined as:

where $P$ is the pooled output,and $p\times q$ is the pooling window size.

The FC layer performs classification,with output $y=W$ . $x+b$ ,where $W$ is the weight matrix, $x$ the input,and $b$ the bias. Finally, softmax converts logits to probabilities:

where ${\zeta }_i$ is the $i$ -th output element,and $n$ is the number of classes.

Some applications of CNNs in traffic scene understanding include early implementations of recognizing traffic signs with 99% accuracy, though the recognition time was relatively long for real-time applications [6]. Modern adaptations have surpassed human performance in tasks like traffic sign recognition [7]. CNNs also enhance free-space detection through data fusion techniques [8] and improve recognition of traffic police gestures [9].

Shortly after the introduction of CNNs in the 1980s, [6] applied fractal texture segmentation for traffic sign detection using a receptive field neural network (NN). The network had an input layer of $32\times 32$ neurons,an output layer with ten neurons,and four hidden layers of $16\times 16,8\times 8,4\times 4$ ,and 30 neurons. It was trained to recognize 9 types of traffic signs from images at 1,2, and 3 meters, achieving 99% accuracy. However, the recognition time of 4 seconds is too long for real-time use. The "RFNN_TSR" dataset includes nine road signs for landmark recognition in outdoor settings at varying distances.

In [7], the traditional CNN architecture was modified to incorporate multi-scale features, achieving an accuracy of 99.17% on the GTSRB dataset, surpassing human performance ( $98.81\mathrm{\%}$ ). Initially,using $32\times 32$ color images, the model achieved 98.97% accuracy, with even randomly generated features yielding a competitive 97.33 .

SNE-RoadSeg [8] integrates surface normal estimation (SNE) with a data-fusion CNN architecture for enhanced free-space detection, showcasing a unique dual-encoder system that merges RGB and surface normal information. This fusion, along with densely-connected skip connections in the decoder, enables precise segmentation. On the KITTI benchmark, it achieves an average precision (AP) of 94.07%.

A novel approach to traffic police gesture recognition is proposed in [9], combining a modified Convolutional Pose Machine (CPM) with a Long Short-Term Memory (LSTM) for temporal feature extraction. Enhanced by handcrafted features like Relative Bone Length and Angle with Gravity, it achieves 91.18% accuracy on the TPGR dataset.

B. R-CNN

In this section, we delve into the R-CNN family of models, which build upon the strengths of CNNs by introducing region-based detection for improved precision in complex scenarios such as traffic monitoring. We will explore the evolution of R-CNN models, beginning with Vanilla R-CNN and progressing through Fast R-CNN, Faster R-CNN, and Mask R-CNN, highlighting their advancements and contributions to traffic scene understanding.

FIGURE 1. Vanilla R-CNN workflow for object detection in a traffic scene: The process starts with identifying a set of proposed regions that could contain objects. Each proposed region is then passed through a pre-trained CNN to extract features, followed by classification using class-specific SVMs. Finally, bounding boxes are refined to enhance localization accuracy. This workflow demonstrates the ability to accurately detect and classify objects such as cars, poles, and trees, achieving precise object localization and high reliability in real-time traffic monitoring applications.

1) VANILLA R-CNN

Vanilla R-CNN [10] extends traditional CNNs by using region proposals and pretrained CNNs for object detection. It generates region proposals to hypothesize object locations, processes each region through a CNN to extract feature vectors, and classifies these vectors with class-specific SVMs and bounding box regressors. This allows R-CNNs to manage object variability and achieve superior detection performance.

Figure 1 illustrates the Vanilla R-CNN process for object detection in a traffic scene. The first step is generating region proposals that may contain objects. If the image is denoted as $I$ ,the set of region proposals can be represented as $R=$ $\left\{r_1,r_2,\dots ,r_n\right\}$ ,where each $r_i$ is a bounding box.

Each region $r_i$ is passed through a pre-trained CNN to extract a feature vector $F_i$ . The CNN maps the region $r_i$ to the feature vector $F_i$ ,represented as $F_i=f\left(r_i\right)$ .

The extracted feature vectors are classified using class-specific linear SVMs. The score for region $r_i$ belonging to class $j$ is $S_{ij}=f_{{\mathrm{SVM}}_j}\left(F_i\right)$ ,where $f_{{\mathrm{SVM}}_j}$ is the SVM for class $j$ .

Finally, the bounding box for each region proposal is refined using a bounding box regressor. This regressor, $g$ , takes the feature vector $F_i$ and the original bounding box $r_i$ as input and outputs a new bounding box $b_i$ ,represented as $b_i=$ $g\left(F_i,r_i\right)$ . The output of R-CNN is a set of bounding boxes with their corresponding class labels, where each bounding box is assigned to the class with the highest SVM score.

R-CNNs have greatly advanced object detection in traffic scenes, surpassing traditional CNNs with accuracy rates of up to 75.6% on the COCO dataset [11]. They excel in detecting pedestrians [12], vehicles [13], and traffic signs [14], [15]. Innovations like cascaded architectures [16], attention mechanisms [17], and hybrid approaches [18] further improve their performance. These advancements contribute to robust traffic scene understanding, aiding the development of automated driving systems [19].

R-CNN, introduced in [10], significantly improves object detection accuracy by combining region proposals with CNNs. It addresses occlusion by using region proposals to better localize objects, even when partially occluded, in contrast to sliding window methods like OverFeat.

In a comparative study, R-CNN outperformed traditional CNN on the COCO dataset,achieving $75.6\mathrm{\%}$ accuracy $(\mathrm{N}=$ 78) compared to $47.7\mathrm{\%}\left(\mathrm{N}=78\right)$ for $\mathrm{CNN}$ . The study,with a $70\mathrm{\%}-30\mathrm{\%}$ training-test split,found a significance p-value of 0.041 [11].

The authors of [12] focus on pedestrian detection, achieving a 23.3% miss rate on the Caltech dataset. They handle occlusion challenges by relying on the model's ability to learn from large datasets instead of explicit occlusion modeling, improving accuracy with more training data.

A method for traffic sign detection using sparse R-CNN [20] is introduced in [18]. On the BCTSDB and TT- $100\mathrm{K}$ datasets,it achieved state-of-the-art performance,with AP50 and AP75 scores of 99.1% and 96.2% for BCTSDB, and 53.1% and 48.7% for TT-100K.

The Bagging R-CNN framework in [17] uses ensemble learning with adaptive sampling to improve object detection in complex traffic scenes. It achieved $58.7\mathrm{\%}$ AP and $83.0\mathrm{\%}$ AP50 using ResNet50, and 63.0% AP and 87.1% AP50 using the Swin-T [21] backbone.

Context R-CNN, as presented in [19], improves stationary surveillance by selecting and storing objects in memory banks by category. This approach enhanced recognition performance on the TJU-DHD-traffic and Pascal VOC datasets, increasing the mean average precision (mAP) by 0.37 compared to conventional methods.

Although Vanilla R-CNN is foundational, its two-stage process is slow and inefficient due to sequential processing of region proposals and classification. This results in high computational complexity and latency, making it unsuitable for real-time applications. Additionally, it struggles with small object detection and relies on selective search, leading to redundant regions, while requiring significant memory and complex training procedures, limiting its practical use.

FIGURE 2. Fast R-CNN procedure for object detection in a traffic scene: The model processes the input image by first extracting features for the entire image using a deep convolutional neural network (Deep ConvNet). RoI projections are mapped onto this shared feature map to generate fixed-size feature vectors using RoI pooling. The resulting RoI feature vectors are passed through fully connected layers to produce two outputs: class probabilities (using a softmax layer for classification) and bounding box regression to refine bounding box coordinates. This enables accurate object detection, such as identifying the “Police Car” class and refining the bounding box parameters in the traffic scene, making it suitable for real-time applications.

2) FAST R-CNN

Fast R-CNN, introduced in [22], improves R-CNN by using a single forward pass of the image through a CNN to extract feature maps. It classifies object proposals and refines their spatial locations directly from shared feature maps, significantly improving training and testing speed. Fast R-CNN trains the VGG16 network [23] nine times faster and tests 213 times quicker than R-CNN, while achieving higher mAP on the PASCAL VOC 2012 dataset and surpassing SPPnet [24] in accuracy.

Figure 2 shows the Fast R-CNN approach for traffic scene object detection. Unlike R-CNN, Fast R-CNN uses a single deep CNN,denoted as $f_{\mathrm{CNN}}$ ,to extract features from the entire image once,producing a feature map $\mathcal{F}=f_{\mathrm{CNN}}\left(I\right)$ .

Region proposals are generated and mapped onto the shared feature map $\mathcal{F}$ . Each region proposal $r_i$ is converted to a fixed-size feature vector $\text{{mathcal{F}}\_{{r}\_i} = {f}\_{text{RoI\_Pooling }}left( {mathcal{F},{r}\_i}right)}$ using RoI pooling.

Fast R-CNN uses a softmax layer for classification, unlike R-CNN’s SVMs. The probability $P_{ij}$ that region $r_i$ belongs to class $j$ is:

where $j$ ranges from 1 to $C$ . For bounding box regression, Fast R-CNN refines the bounding box using predicted offsets $\delta x,\delta y,\delta w,\delta h$ ,calculated as:

The final output is a set of refined bounding boxes with class labels. Each bounding box is associated with the highest probability from the softmax layer.

Fast R-CNN has been successfully applied in various traffic scene tasks, enhancing detection and classification. It detects road surface signs [25], counts and identifies vehicles in challenging scenarios [26], and improves monitoring at intersections [27]. The technology also enables simultaneous detection of pedestrians and cyclists, excelling on urban datasets [28]. Additionally, it has been adapted for event-based vehicle classification and counting, demonstrating its versatility in dynamic traffic environments [29].

The authors of [28] propose a unified method for concurrent detection of pedestrians and cyclists using a novel UB-MPR detection proposal and a Fast R-CNN-based model. Tested on the Tsinghua-Daimler dataset, it achieves a recall rate of $96.5\mathrm{\%}$ at an $\mathrm{IoU}$ threshold of 0.5 . The method addresses occlusion effectively by focusing on upper body detection, ensuring accurate detection even with partial occlusion.

In [30], a framework combining deformable part models (DPMs) with CNNs and region proposal networks (RPNs) accelerates Fast R-CNN. Tested on the KITTI car benchmark, it shows $70\mathrm{\%}$ overlap for true positives across Easy,Moderate,and Hard settings,with up to $12\mathrm{x}$ speed improvement and comparable precision, especially in PASCAL VOC and KITTI assessments.

In [27], a road user monitoring system for intersections is presented, combining a GMM-based DL approach with geometric warping. Integrated with Fast R-CNN, it processes 0.92s and $0.99\mathrm{s}$ faster on the MIT and Jinan datasets, respectively, with fewer misses in tracking and classification.

The study in [31] proposes a joint detection framework for pedestrians and cyclists using Fast R-CNN, incorporating techniques like difficult case extraction, multi-layer feature fusion, and shared convolution layers. This deeper architecture outperformed its counterpart, achieving 4.3% and $5.6\mathrm{\%}$ higher accuracy in pedestrian and cyclist detection, respectively.

In [29], an event-based object detection system using Fast R-CNN with hyperparameter optimization on modified Stanford car and Myanmar cars datasets is introduced. It achieves accurate vehicle classification and counting in real-time event video streaming, with improved accuracy for weddings and precise learning rate assessments on the Myanmar Cars dataset.

Fast R-CNN is used in [32] (referred to as "AllLightR-CNN" in our work) to detect moving vehicles in various conditions, such as low light, long shadows, cloudy weather, and dense traffic. It achieves an average computation time of 0.59 seconds with high detection rates, including 98.44% recall, $94.20\mathrm{\%}$ accuracy,and $90\mathrm{\%}$ precision in both day and night modes. The dataset ("AllLightRCNN_DS" in our work) includes 3975 frames from four YouTube videos, annotated for vehicle detection and classification, featuring occlusion and varying times of day and weather conditions.

Fast R-CNN improves speed over R-CNN by processing the entire image in a single pass but still relies on time-consuming region proposals, limiting real-time performance. While RoI pooling speeds up processing, it can introduce quantization errors, reducing accuracy, especially for small objects. Additionally, relying on external region proposal methods like Selective Search hinders real-time capabilities. Fast R-CNN also requires large labeled datasets and struggles with high-resolution images, where detailed feature extraction is critical.

3) FASTER R-CNN

Faster R-CNN, first introduced in [33], improves upon its predecessors, R-CNN and Fast R-CNN, by incorporating a Region Proposal Network (RPN). This RPN shares full-image convolutional features with the detection network, allowing for nearly cost-free region proposals.

Figure 3 illustrates the use of Faster R-CNN for object detection in a traffic scene. Faster R-CNN comprises two main modules: a deep fully convolutional network for proposing regions and a Fast R-CNN detector that classifies these regions. Together, these modules form a unified network for efficient object detection in complex environments like traffic scenes.

Given an input image, the first step in Faster R-CNN is to pass it through several convolutional and max pooling layers to produce a shared feature map. If the input image is denoted as $X$ ,the convolution operation can be represented as:

where $\mathcal{F}$ is the resulting feature map.

The RPN takes the shared feature map $\mathcal{F}$ and outputs a set of rectangular object proposals, each with an objectness score. The RPN is fully convolutional and simultaneously predicts multiple region proposals at each location. If $f_{\mathrm{RPN}}\left(\mathcal{F}\right)$ represents the RPN operation,the output $\mathrm{Y}$ can be represented as:

where $P_i$ is the $i$ -th proposed region and $s_i$ is the corresponding score.

The proposed regions are reshaped using an RoI pooling layer to provide a fixed-size input to the FC layers. This process is the same in both Fast R-CNN and Faster R-CNN, with the key difference being the source of the regions: external algorithms in Fast R-CNN and an internal RPN in Faster R-CNN. The reshaping by RoI pooling can be mathematically expressed as:

where $R_i$ represents the reshaped region derived from each proposed region $P_i$ and the feature map $\mathcal{F}$ .

Finally, the reshaped regions are fed into a sequence of FC layers that output the class probabilities and bounding box coordinates. If $f_c\left(R_i\right)$ represents the classification operation, the overall output of Faster R-CNN can thus be represented as follows:

Faster R-CNN has advanced traffic scene understanding across several applications, improving environmental perception [34], optimizing traffic sign detection [35], and enhancing recognition of police gestures [36]. It has also refined pedestrian detection [37], boosted traffic surveillance [38], and enabled accurate vehicle categorization in traffic surveys [39]. These advancements improve performance in diverse conditions, aiding autonomous driving [40] and supporting real-time traffic analysis in smart cities [41].

An improved Faster R-CNN for small object detection is proposed in [42], specifically targeting small traffic signs in the TT100K dataset. It achieves a recall rate of $90\mathrm{\%}$ and an accuracy rate of $87\mathrm{\%}$ . The method addresses occlusion by using multi-scale convolution feature fusion and improved non-maximum suppression (NMS), enhancing the detection of small and partially occluded objects.

In [39], Faster R-CNN is used for vehicle detection in traffic surveys, chosen over SSD [43] and YOLO [44] for its higher accuracy despite slower speed. The authors highlight the advantages of DL over traditional methods, achieving over $87\mathrm{\%}$ accuracy in vehicle detection,queue length estimation, and vehicle type classification, even in untrained environments.

FIGURE 3. Faster R-CNN workflow for object detection in a traffic scene: The process starts by passing the image through several convolutional layers to generate a shared feature map. The feature maps are then processed by a region proposal network, which produces a set of region proposals with corresponding objectness scores. These proposed regions are reshaped using RoI pooling to ensure a consistent input size for the fully connected layers. Finally, the reshaped regions are classified into specific object categories, such as sign poles and police cars, and adjusted for accurate bounding boxes. localization, resulting in precise detection of various elements in the traffic scene.

In [45], Faster R-CNN is enhanced for object detection using hard negative sample mining and a two-channel feature network. By treating complex multi-classification tasks as binary classification, the modified approach achieves a 5% accuracy improvement on the KITTI dataset.

The authors of [40] propose an enhanced Faster R-CNN for traffic sign detection, incorporating feature pyramid fusion, deformable convolution, and ROI Align. Tested under various conditions,it achieved mAP scores of $92.6\mathrm{\%}$ in sunny, $90.6\mathrm{\%}$ at sunset,and $86.9\mathrm{\%}$ on rainy days,outperforming SSD [43], YOLOv2 [46], YOLOv3 [47], and YOLOv5 [48] in low-light and rainy conditions, proving effective for autonomous driving.

In [49], an enhanced Faster R-CNN with ResNet50-D, an attention-guided context feature pyramid network (ACFPN), and AutoAugment technology is proposed for traffic sign detection. Benchmarking against methods like SSD [43] and YOLOv3 [47], it achieved 29.8 FPS and 99.5% mAP on the CCTSDB dataset, surpassing other state-of-the-art methods,with competitive results on the TT100K dataset.

In [50], a correlation model analyzes haze's impact on traffic sign detection and sight distance, using a synthesized GTSDB dataset. The Faster R-CNN model, post-dehazing, achieved 95.11% detection accuracy. Results show that haze intensity inversely affects sight distance and detection, with accuracies of over 93% at 300 meters in light haze, 88%-93% at 100 meters in haze,and $85\mathrm{\%}-88\mathrm{\%}$ at 50 meters in dense haze.

In [41], the authors discuss the Intelligent Transportation System (ITS)-oriented Information Acquisition Models (IAMs), using the Mirror Traffic dataset and Internet of Things (IoT) to predict traffic conditions and adjust signals in real-time. By comparing Faster R-CNN to R-CNN in a DL context, they found that Faster R-CNN, with an 85.10% recall and 86.79% accuracy, outperforms R-CNN by 6.20%.

Faster R-CNN enhances speed by integrating region proposal generation through an RPN and improves occlusion handling, enabling better detection of partially obscured or overlapping objects. Despite these advances, it still faces challenges with real-time processing due to the computational demands of the RPN, high memory usage, and the need for extensive training data. The model struggles with small, overlapping, or occluded objects, and its complexity makes implementation and tuning difficult, limiting its use in data-scarce scenarios.

4) MASK-R-CNN

Mask R-CNN [51] extends Faster R-CNN by adding a branch for object masks alongside class labels and bounding-box offsets. It achieves fine spatial layout extraction via pixel-to-pixel alignment, addressing limitations in Fast and Faster R-CNN. Retaining a two-stage approach, the first stage uses an RPN to propose regions, while the second stage predicts classes, bounding-box offsets, and binary masks for each RoI. RoI Align ensures precise feature alignment, improving segmentation accuracy. By predicting classification, regression, and segmentation in parallel, Mask R-CNN streamlines the multi-stage pipeline, offering a powerful solution for segmentation tasks.

Figure 4 shows Mask R-CNN applied to instance segmentation in a traffic scene. Key components unique to Mask R-CNN are highlighted, excluding those shared with Faster R-CNN (e.g., the backbone, RPN, and bounding-box regression). The focus is on its distinctive elements: the RoI Align operation and the mask prediction process.

After the RPN identifies potential object bounding box locations in the image, RoI Align warps features from the feature map to a fixed-size representation for each RoI without quantization:

FIGURE 4. Mask R-CNN procedure for instance segmentation in a traffic scene: The input image is first processed through an RPN to identify regions of interest. These regions are then refined using the RoI Align operation, which ensures precise feature extraction by avoiding quantization effects, leading to more accurate segmentation. The refined features are passed through fully connected layers for class prediction and bounding box regression. Subsequently, the mask prediction process generates detailed binary segmentation masks for each instance using convolutional layers, producing accurate pixel-level masks. This approach provides high-resolution masks for various objects within the traffic scene, such as vehicles, pedestrians, and traffic signs, enabling precise instance segmentation.

where $w_{\mathrm{ROI}}$ and $h_{\mathrm{ROI}}$ represent the width and height of the RoI,and $w_{\text{pooled }}$ and $h_{\text{pooled }}$ represent the width and height after RoI pooling, respectively.

The mask prediction process uses a Mask Head that generates binary masks for each RoI. In Equation 11, $\text{{f}\_{text{Conv\_2D }left( cdot right) }}$ is the 2D convolution operation on the input features. In Equation 12, ${\zeta }_{\text{Mask }}$ are raw mask predictions before the sigmoid function,and $P\left({\zeta }_{\text{Mask }}\right)$ gives final pixel-level mask probabilities within the RoI:

Mask R-CNN has been applied to various tasks, including floodwater detection on roads [52], traffic sign detection and recognition [53], and train safety through improved obstacle identification [54]. It also supports urban traffic management via vehicle contour detection and tracking [55], and enables accurate vehicle counting to manage congestion [56]. Comparative studies confirm its superior performance in vehicle detection and classification compared to other models [57].

In [52], a Mask R-CNN-based method for floodwater detection achieves 99.2% classification accuracy and 93.0% segmentation precision on the IDRF dataset [58], outperforming a prior approach [59]. For traffic sign detection and recognition, [53] uses a two-phase method with Mask R-CNN for shape-based detection and Xception [60] for classification on 11,074 Taiwanese traffic signs, achieving $98.45\mathrm{\%}$ precision for triangular and 99.73% for circular signs, surpassing YOLOv5 [61].

The ME Mask R-CNN method [54] improves automated train safety by integrating SSwin-Le Transformer, ME-PAPN, and multiscale enhancements, achieving a 91.3% mAP on the TrainObstacle dataset, 11.1% higher than Mask R-CNN, and an average detection rate of 4.2 FPS. It improves small-target detection by 19.35%, though gains for large and occluded targets are limited due to dataset characteristics.

A comprehensive comparison [57] of Faster R-CNN, Mask R-CNN, and ResNet-50 (R-CNN) on the 3,200-image RCNNs_Detection dataset (cars and jeeps from Kaggle) shows that Faster R-CNN and Mask R-CNN exceed 80% detection accuracy, while ResNet-50 achieves over 75%. This demonstrates their effectiveness in vehicle detection, classification, and counting.

Mask R-CNN extends Faster R-CNN with a mask prediction branch, enabling instance segmentation and facilitating the detection of occluded objects by distinguishing overlapping instances. However, this extra branch increases computational and memory demands, complicating real-time use and deployment on resource-constrained devices. The model also requires substantial labeled data, longer training times, and can struggle with small objects and complex scenes. Its increased complexity makes implementation, tuning, and debugging more challenging, particularly for custom applications.

C. YOLO

YOLO is a fast and efficient real-time object detection system that predicts detections in a single pass, unlike R-CNN methods relying on region proposals. Introduced in [44], it treats detection as a regression problem, dividing the image into a grid where each cell predicts bounding boxes, confidence scores, and class probabilities. YOLO generalizes well across domains but struggles with precise localization, especially for small objects. Fast YOLO, the then-fastest general-purpose detector, is also introduced.

FIGURE 5. YOLO object detection in a traffic scene featuring a police car: The input image is initially divided into an SxS grid, with each cell predicting bounding boxes, confidence scores, and class probabilities. This process culminates in a final detection display that accurately identifies and localizes the police car and other objects within the scene. A comprehensive legend highlights key objects of interest, providing a clear and detailed overview of the detected items. This real-time object detection approach combines speed and accuracy, making it highly effective for dynamic traffic monitoring applications.

Figure 5 illustrates YOLO applied to a traffic scene. An input image $I$ is divided into an $S\times S$ grid. Each grid cell detects objects if their center falls within it. A convolutional network processes the image and outputs a tensor of shape $S\times S\times \left(B\times 5+C\right)$ ,where $B$ is the number of bounding boxes per cell,and $C$ is the number of classes.

Bounding boxes are defined by(x,y,w,h,s),where(x,y) are the center coordinates relative to the grid cell, $w$ and $h$ are relative to the image size,and $s$ is the confidence score, indicating the likelihood and accuracy of the box containing an object.

Each grid cell predicts $C$ conditional class probabilities, $P\left({\mathrm{c}}_i\mid \hat{o}\right)$ ,for the detected object $\hat{o}$ belonging to class $c_i$ . The final prediction combines these probabilities with the confidence score:

YOLO applies Non-Maximum Suppression (NMS) to remove redundant overlapping boxes. Predictions are sorted by confidence score,and overlapping boxes (e.g.,IoU > 0.5) with lower scores are removed.

YOLO’s operation is summarized as $o=f_{\mathrm{NMS}}\left(f_{\mathrm{Conv}}\left(I\right)\right)$ , where $o$ is the final prediction, $f_{\text{Conv }}$ represents convolutional layers,and $f_{\mathrm{NMS}}$ applies NMS.

YOLO has been adapted for traffic flow counting [62], traffic light detection [63], and traffic sign recognition [64], [65]. It effectively detects pedestrians and vehicles [66] and operates under varied lighting and weather conditions [67], [68]. Continuous improvements enable better small-target detection and high-resolution video processing [69], [70]. Additionally, YOLO versions support license plate identification [71] and have been compared across models for traffic sign detection and vehicle classification in challenging environments [72], [73].

YOLO has seen numerous enhancements and iterations since its inception. The following presents the primary YOLO versions, accompanied by historical insights and a review of relevant scholarly literature.

1) YOLOV1 (YOLO)

Debuted in 2016, this original YOLO model was ground-breaking as it treated object detection as a singular regression problem, enabling it to predict bounding box coordinates and class probabilities from an image in a single pass [44].

The importance of Traffic Light Detection (TLD) for intelligent vehicles and Driving Assistance Systems is highlighted in [63], which applies YOLO to the daySequence1 from the LISA Traffic Light Dataset. The study achieves a 90.49% AUC, a 50.32% improvement over the previous best using Aggregated Channel Features (ACF), and a 58.3% AUC comparable to the ACF configuration. This underscores TLD's critical role in enhancing self-driving car functionality.

2) YOLOv2

Launched in 2017, YOLOv2 [46] introduced major improvements, including detection of over 9000 object categories, the "Darknet-19" architecture, anchor boxes for better bounding box prediction, and multi-scale training. By combining detection-labeled data from COCO with classification data from ImageNet [74], the authors enabled joint classification and detection, creating YOLO9000 [46], capable of detecting a vast range of categories.

An optimized pedestrian and vehicle detection algorithm based on YOLOv2 is introduced in [66], which improves accuracy while maintaining efficiency. Comparative results on the KITTI dataset demonstrate its real-time capability, outperforming Faster R-CNN and YOLO V2 with 45% accuracy for pedestrians and $61.34\mathrm{\%}$ for vehicles at 22 FPS.

3) YOLOv3

Unveiled in 2018, YOLOv3 employed three distinct sizes of anchor boxes for predictions across three scales. It utilized a deeper architecture, "Darknet-53," and expanded its object category detection capabilities. Additionally, it adopted three different sizes of detection kernels $\left(13\times 13,26\times 26,52\times 52\right)$ to identify objects of varying dimensions [47].

A YOLOv3-based traffic sign recognition system introduced in [64] achieves a $92.2\mathrm{\%}$ mAP for detection on the GTSDB dataset and 99.6% classification accuracy on the GTSRB dataset using a CNN-based classifier. It outperforms Faster R-CNN in both detection accuracy and frame rate, processing images at approximately 9.87 FPS.

4) YOLOv4

Released in 2020, YOLOv4 incorporated numerous improvements over its predecessors. It integrated features such as the "CSPDarknet53-PANet-SPP" architecture, PANet, and SAM block. Additionally, it employed the Complete IoU (CIoU) loss and the Mish activation function, aiming to enhance both speed and accuracy [75].

In [76], traffic sign detection and recognition for smart vehicles are explored using YOLOv4 and YOLOv4-tiny [75] integrated with Spatial Pyramid Pooling (SPP). Results show Yolo V4_1 (with SPP) achieving 99.4% accuracy and 99.32% mAP, while Yolov3 [47] SPP attains 98.99% mAP. These findings indicate that SPP enhances model performance.

To address environmental challenges like light intensity, extreme weather, and distance, TSR-YOLO [68], based on YOLOv4-tiny, incorporates Better-ECA (BECA), dense SPP networks, and k-means++ clustering for optimal prior boxes. On the CCTSDB2021 dataset,it achieves 96.62% accuracy, 79.73% recall,an 87.37% F1-score,and a 92.77% mAP, improving over YOLOv4-tiny while maintaining 81 FPS.

A novel semi-automatic method, combining a modified YOLOv4 and background subtraction, is introduced in [77] for unsupervised object detection in surveillance videos. It significantly increases mAP and outperforms state-of-the-art results on the CDnet 2014 and UA-DETRAC datasets, achieving $97.4\mathrm{\%}$ precision compared to YOLOv3’s $89\mathrm{\%}$ and YOLOv4's 90.4% on the Street corner at night scenario.

5) YOLOv5

Introduced in 2020, YOLOv5 [61], developed independently, is not an official continuation by the original YOLO creators. It features a modified "CSPDarknet53" backbone with architectural optimizations to improve speed and real-world applicability. Its naming sparked controversy, as it was not created by the original authors.

To improve vehicle detection in traffic surveillance videos, [69] proposes an enhanced YOLOv5s model with a small target detection layer and Atrous SPP (ASPP) for multi-scale context, achieving 93.7% precision, 94.2% recall, and 93.9% mAP@0.5—improvements of 0.8%, 1.9%, and 2.3% over the original YOLOv5s, reducing missed and false detections.

Ghost-YOLO [70] is a lightweight model for traffic sign detection using the C3Ghost module to replace YOLOv5's feature extraction. It achieves $92.71\mathrm{\%}\mathrm{mAP}$ while reducing parameters by 91.4% and computations by 50.29%, balancing speed and accuracy for real-world use.

6) YOLOv6

Introduced in September 2022, YOLOv6 boasts an efficient design comprising a backbone with RepVGG [78] or the newly introduced "CSPStackRep" blocks, a Path Aggregation Networks (PAN) topology neck, and a decoupled head with a hybrid-channel strategy. It uses advanced quantization techniques, such as post-training quantization and channel-wise distillation, leading to swifter and more precise detectors [79].

A license plate identification algorithm is outlined in [71] based on the YOLOv6 convolution model, enhancing efficiency with a 94.7% precision rate for location and a proposed BLPNET(VGG-19-RESNET-50) model achieving 100% F1- score in character recognition, leading to reduced costs and improved traffic management effectiveness.

7) YOLOv7

Released in July 2022, YOLOv7 set new object detection benchmarks, excelling in speed (5)-160 FPS) and accuracy. Trained solely on the MS COCO dataset without pre-trained backbones, it introduced architectural modifications and "bag-of-freebies" to boost accuracy without sacrificing inference speed, though training time increased [80].

An enhanced YOLOv7-WCN network for traffic sign detection [81] improves accuracy from 85.5% to 89.0% by integrating Horblock modules with convolutional layers for efficient mapping, a normalization-based attention module (NAM), and replacing CIoU loss with Wasserstein distance loss [82].

8) YOLOv8

Unveiled in January 2023, YOLOv8 [83] offers five scaled versions: YOLOv8n (nano), YOLOv8s (small), YOLOv8m (medium), YOLOv81 (large), and YOLOv8x (extra large), using a backbone similar to the one used in YOLOv5 but with some modifications on the cross-stage partial (CSP) layer. This iteration supports a wide range of computer vision tasks, including object detection, segmentation, pose estimation, tracking, and classification.

To address road accidents caused by human error, [72] proposes a traffic sign detection method using YOLOv5s6 [61] and YOLOv8s [83]. Testing on TT100k, TWTS, and a hybrid dataset shows YOLOv8s outperforms YOLOv5s6, achieving an mAP@. 5 of $76.2\mathrm{\%}$ compared to $65\mathrm{\%}$ on the hybrid dataset.

Recent studies have compared object detection algorithms across diverse environments and datasets. Reference [84] evaluated Faster R-CNN, YOLOv3, and YOLOv4 for aerial car detection on Stanford and PSU datasets, highlighting the impact of dataset characteristics and parameters like input size and learning rate on accuracy. On the PSU dataset, YOLOv3 and YOLOv4 achieved AP scores of 0.965, outperforming Faster R-CNN (0.739).

A broader evaluation of SSD, Faster R-CNN, and YOLO versions (YOLOv8, YOLOv7, YOLOv6, YOLOv5) on nine datasets (referred to as "ShokriCollection_DS" in our work) featuring varied road challenges was conducted in [73]. YOLO, particularly YOLOv7, excelled with over 95% detection accuracy and $90\mathrm{\%}$ Overall Accuracy (OA) for vehicle classification, despite differing computation times. Metrics were averaged across high-quality datasets from YouTube and Kaggle.

Another comparison [85] shows that YOLOv6 (73.5 mAP) and YOLOv8 (71.8 mAP) significantly outperform DETR ( $49.9\mathrm{mAP}$ ) when benchmarking various state-of-the-art object detectors and exploring large computer vision models as image annotators on the RSUD20K dataset for road scene understanding in autonomous driving.

YOLOv1 [44] and YOLOv2 [46] struggled with occlusions due to coarse feature extraction, limiting the detection of overlapping objects. YOLOv3 [47] improved this with multi-scale detection, and YOLOv4 [75] enhanced occlusion handling using feature pyramid networks. Despite these advances, challenges persist. CCW-YOLO [86] improved detection in dense scenes with a lightweight convolutional layer and C2f module,while HCLT-YOLO [87] used a hybrid CNN and transformer to reduce false alarms and missed detections.

YOLO is fast and efficient for real-time object detection due to its single-shot approach, processing an entire image in one pass. However, this design can struggle with detecting small objects, as the grid-based prediction may lack the precision needed for finer details. YOLO also has difficulty handling overlapping instances, where objects are close together, leading to potential inaccuracies. Additionally, its emphasis on speed may trade off some accuracy, particularly in complex scenes with multiple objects or intricate backgrounds, where precise localization and classification become more challenging.

D. ViT

ViT, introduced in [88], applies transformers to image classification by processing images as patch sequences. Using self-attention, it captures complex dependencies and prioritizes visible features and context, more effectively handling occlusions by inferring hidden objects. Unlike CNNs with localized receptive fields, ViT captures long-range dependencies across the entire image, marking a significant shift in image processing.

Figure 6 illustrates the application of ViT to classification in a traffic scene. The initial step in ViT is Image Patching which involves splitting an image into a series of fixed-size patches. These patches are then flattened and linearly embedded. Additionally, position embeddings are added to retain positional information:

where $e_0$ is the initial input embedding to the transformer, $x_p^i$ is the $i$ -th image patch, $E$ is the patch embedding projection, $E_{\text{pos }}$ are the position embeddings, $x_{\text{class }}$ is a learnable embedding that serves as a representation of the entire image,and $N$ is the total number of patches.

The embedded patches then pass through a series of Transformer encoder layers. Each layer comprises two main parts: a multi-headed self-attention mechanism (MHSA) and a position-wise feed-forward network (FFN).

The MHSA is a self-attention mechanism that allows the model to weigh the importance of different patches when processing each patch:

where $Q,K,V$ are the queries,keys,and values,respectively, computed from the input embeddings,and $d_k$ is the dimension of the keys.

The attention output is processed through multiple "heads," and the results from these heads are concatenated, represented by the symbol $\parallel$ :

where the $i$ -th head is computed as:

and $W_i^Q,W_i^K,W_i^V$ ,and $W^O$ are learned parameter matrices, and $n$ is the number of attention heads.

The position-wise FFN consists of two linear transformations with a ReLU activation in between:

where $x$ is the input to the FFN, $W_1$ and $W_2$ are weight matrices,and $b_1$ and $b_2$ are bias vectors.

After passing through the Transformer layers, the class token's output (from the final layer) is used to predict the class of the image via a simple linear layer:

where $z_{\text{class }}^L$ is the output corresponding to the class token from the last Transformer layer, $W^C$ is the output projection matrix,and $o$ is the output.

FIGURE 6. Application of ViT in classifying a traffic scene with a crosswalk: The input image is divided into fixed-size patches, which are then flattened and linearly projected into an embedding space. Position embeddings are added to these patch embeddings to retain spatial relationships, along with a class embedding to represent the entire image. The combined embeddings are sequentially processed through the Transformer encoder, involving multiple layers of multi-head self-attention and feed-forward networks. The class token output from the final Transformer layer is passed through an MLP head to predict the class label, in this case, 'crosswalk'

Some applications of ViTs include detecting rain and road surface conditions [89], predicting pedestrian crossing intentions [90], identifying critical traffic moments [91], and detecting unusual traffic scenarios [92].

A cost-effective method for detecting rain and road conditions using ViTs and a Spatial Self-Attention network is presented in [89], achieving F1-scores of 91.13% for rain and 92.10% for road conditions. Adding a sequential detection module improved accuracy to 96.74% and 98.07%, respectively. The study's dataset, referred to as "ViT_DS" in our work, includes 10,000 freeway images from CCTV cameras in Orlando, Florida, labeled for 3 rain levels and 2 road condition levels.

Action-ViT [90] integrates multimodal data-including visual cues, poses, bounding boxes, and action annotations-and employs tailored data processing for each modality, enhancing pedestrian crossing intention prediction and achieving a $90.2\mathrm{\%}$ F1-score on the JAAD dataset,with ablation studies confirming improvements in temporal modeling and feature fusion.

ViT-TA [91] is a custom ViT that achieves 94% accuracy in detecting critical moments at Time-To-Collision (TTC) $\le$ 1s on the Dashcam Accident Dataset (DAD). It classifies critical traffic situations and uses attention maps to highlight probable causes, systematically enhancing automated vehicle safety by generating reliable safety scenarios.

Vit-L [92] detects scenario novelty in traffic using infrastructure images and a triplet autoencoder trained on 70,000 traffic scene and graph pairs in Germany. Enhanced by expert domain knowledge and ViTs, it uses Angle-Based Outlier Detection (ABOD) in the latent space, achieving a 95.6% AUC. The dataset, referred to as "Wurst_DS" in our work and detailed in [93], comprises highway images for outlier model fitting.

ViTs excel at capturing long-range dependencies in images, allowing for a more holistic understanding of visual data. However, they require large amounts of data and substantial computational power to achieve high performance, making them less accessible in data-limited scenarios. ViTs can struggle with generalizing from smaller datasets, often leading to overfitting or suboptimal results. Additionally, they may be less efficient than CNNs for lower-resolution images, where the advantage of capturing long-range dependencies is diminished, and the computational overhead becomes more pronounced.

E. DETR

DETR, introduced in [94], is an innovative model for object detection that leverages the Transformer architecture to streamline the process into an end-to-end framework. By treating object detection as a direct set prediction problem, DETR eliminates the need for hand-designed components like non-maximum suppression and anchor generation. The model employs a combination of a CNN for feature extraction and a Transformer for decoding these features into bounding box predictions and class labels in a single forward pass. Leveraging self-attention, DETR models complex relationships between objects and their context, making it particularly effective at detecting and localizing partially occluded objects by interpreting visible fragments within the overall scene. This unified transformer-based approach marks a significant advancement in handling occlusions and simplifying object detection.

FIGURE 7. DETR object detection in a traffic scene: The process begins with a CNN extracting image features, which are then enhanced with positional encodings to preserve spatial information and processed through a transformer encoder. The encoder employs several layers of self-attention and FFNs to refine these features for improved detection accuracy. The transformer decoder uses a fixed set of learned object queries, combined with the encoded features, to generate predictions for possible objects, including their classes and bounding boxes. Four FFNs are employed to finalize classifications and bounding box coordinates, effectively highlighting detected objects in the scene.

Figure 7 depicts the application of DETR to object detection in a traffic scene. DETR starts with feature extraction through which given an input image $I$ ,a $f_{\text{ConvNet }}$ is used to extract a feature map $\mathcal{F}$ where $\mathcal{F}=f_{\text{ConvNet }}\left(I\right)$ , and $f_{\text{ConvNet }}$ is a CNN.

Positional encodings (PEs) are then added to the feature map $\mathcal{F}$ to preserve spatial information,resulting in ${\mathcal{F}}^{\prime }=$ $\mathcal{F}+PE$ ,where ${\mathcal{F}}^{\prime }$ denotes the feature map with positional encoding added.

At the next step, the Transformer encoder processes this enhanced feature map ${\mathcal{F}}^{\prime }$ through several layers of self-attention and FFNs,resulting in ${\mathcal{F}}_z=f_{\text{Encoder }}\left({\mathcal{F}}^{\prime }\right)$ ,where ${\mathcal{F}}_z$ is the encoded feature representation.

The Transformer decoder uses a set of fixed learned object queries $\mathcal{Q}$ and the encoded features ${\mathcal{F}}_z$ to generate predictions:

where $\mathcal{Q}$ is the set of $N_q$ learned object queries,and $q_i$ represents the $i$ -th query embedding. The output of the decoder is $o=f_{\text{Decoder }}\left({\mathcal{F}}_z,\mathcal{Q}\right)$ ,where $o$ contains the predictions for potential objects, represented as classes and bounding boxes.

The outputs of the Transformer decoder are then processed to yield a fixed-size set of predictions, irrespective of the number of objects in the image. This is represented as:

where ${\hat{c}}_i$ and ${\hat{b}}_i$ are the predicted class and bounding box coordinates for the $i$ -th object,and $N$ is the number of predictions.

The loss function is a crucial part of training DETR, incorporating a unique bipartite matching loss to match predicted and ground truth objects, along with classification and bounding box regression losses.

Bipartite matching is used to find the optimal permutation of predicted objects $\sigma$ that minimizes the matching cost, as described in [94]:

where ${\sum }_N$ is the set of all permutations of $N$ elements, $y_i$ is the ground truth,and ${\hat{y}}_i$ is the prediction. Here, $f_{\text{cost }}$ is the cost function that measures the difference between the ground truth objects $y_i$ and the predicted objects ${\hat{y}}_{\sigma \left(i\right)}$ .

The loss function $\mathcal{L}$ combines the costs of classification and bounding box prediction:

(23)

where $c_i^{\ast }$ and $b_i^{\ast }$ are the true class and bounding box of the $i$ -th object,respectively,and ${\lambda }_{\mathrm{cls}}$ and ${\lambda }_{\text{bbox }}$ are weighting factors that determine the relative importance of the classification loss ${\mathcal{L}}_{\text{cls }}$ and the bounding box loss ${\mathcal{L}}_{\text{bbox }}$ . The loss function $\mathcal{L}$ is minimized during training to ensure that the predicted outputs closely match the ground truth annotations. This approach helps achieve accurate object detection by focusing on both the correctness of the predicted class and the accuracy of the bounding box localization.

During training, DETR minimizes this loss function to learn the parameters that result in the best predictions of object classes and bounding boxes, tailored to match the true objects in the image as closely as possible. This streamlined approach of direct set prediction and loss minimization via bipartite matching distinctly sets DETR apart in the field of object detection.

DETR's applications are diverse, including enhancements for detecting traffic signs of various sizes [95], recognizing small or weather-affected signs [96], and accelerating model training [97]. Additionally, DETR enhances object detection for autonomous driving by effectively aligning objects with their respective scenes [98].

An innovative approach to traffic sign detection, DSRA-DETR [95], emphasizes enhanced multiscale detection performance through modules that aggregate features across scales, effectively reducing feature noise, preserving low-level features, and boosting the model's ability to recognize objects at various sizes. This results in significant improvements in detection accuracy with impressive APs of 76.13% and 78.24% on GTSDB and CCTSDB datasets, respectively.

MTSDet [96] enhances traffic sign detection by using an Attention Mechanism Network (AMNet) and a Path Aggregation Feature Pyramid Network (PAFPN) for multi-scale feature fusion. It excels at detecting small or weather-affected signs, achieving mAP scores of 92.9% on GTSRB and 94.3% on CTSD.

In [97], a Spatially Modulated Co-Attention (SMCA) mechanism improves DETR by focusing co-attention near initial box estimates and integrating multi-head, scale-selection attention. This yields $45.6\mathrm{mAP}$ in 108 epochs, surpassing DETR’s original $43.3\mathrm{mAP}$ in 500 epochs, as verified by extensive ablation studies on COCO.

DetectFormer [98] improves autonomous driving object detection by incorporating a ClassDecoder and a Global Extract Encoder (GEE) to enhance category sensitivity and scene alignment. With data augmentation and attention mechanisms, it achieves AP50 and AP75 scores of 97.6% and 91.4%, respectively, on the BCTSDB dataset.

DETR simplifies object detection by eliminating the need for region proposals and streamlining the process with a transformer-based architecture. However, it requires extensive training data to perform well and is computationally intensive, making it challenging to deploy in resource-constrained environments. DETR can also be slower to converge during training, requiring more epochs to reach optimal performance. Additionally, it struggles with detecting small objects in cluttered scenes, where the lack of region proposals can lead to less precise localization and classification.

F. GRAPH NEURAL NETWORK (GNN)

GNNs are a powerful tool for traffic scene understanding, representing road networks as graphs and capturing spatial-temporal relationships. They enable precise analysis of vehicle trajectories, pedestrian movements, and interactions, aiding tasks like congestion prediction, collision avoidance, and adaptive signal control. By leveraging graph-based methods, GNNs enhance real-time decision-making in intelligent transportation systems, contributing to safer and more efficient urban mobility.

1) GCN

GCN, first introduced in [99], is a neural network designed for graph-structured data, extending the concept of convolution from grid-like data (e.g.,images) to graphs. A graph $\mathcal{G}=$ (V,E)consists of nodes $V\left(\left|V\right|=N\right)$ and edges $E$ , represented by an adjacency matrix $A$ ,where $A_{ij}=1$ if an edge exists between nodes $i$ and $j$ ,and 0 otherwise. Each node $v_i\in V$ has a feature vector ${\mathcal{F}}_i\in {\mathbb{R}}^d$ ,and the node features collectively form a matrix $\mathcal{F}\in {\mathbb{R}}^{N\times d}$ ,where $d$ is the number of features per node.

The core idea of a GCN is to perform a convolution-like operation on a graph. The graph convolution for a single layer is expressed as:

where ${\mathcal{F}}^{\left(l\right)}\in {\mathbb{R}}^{N\times F^{\left(l\right)}}$ is the input feature matrix at layer $l$ , $W^{\left(l\right)}\in {\mathbb{R}}^{F^{\left(l\right)}\times F^{(l+1)}}$ is the weight matrix, $\sigma$ is a non-linear activation function (e.g.,ReLU),and $\hat{A}$ is the normalized adjacency matrix with self-loops.

The normalized adjacency matrix $\hat{A}$ is defined as:

where $\tilde{A}=A+I_d$ (adjacency matrix with self-loops),and $\mathrm{\Delta }$ is the degree matrix of $\tilde{A}$ with diagonal elements ${\mathrm{\Delta }}_{ii}=\sum _j{\tilde{A}}_{ij}$ .

A typical GCN model has multiple layers. For instance, a two-layer GCN is:

where $X$ is the input feature matrix, $W^{\left(0\right)}$ and $W^{\left(1\right)}$ are the weight matrices, $\sigma$ is the activation function,and $O$ represents the final output, e.g., class probabilities for node classification.

The GCN is trained by minimizing the cross-entropy loss:

where $\mathcal{Y}$ is the set of labeled nodes, $Y$ is the label matrix,and $C$ is the number of classes.

By iteratively updating $W^{\left(l\right)}$ using gradient descent,the GCN learns features from the graph structure and node attributes for the target task.

GCNs excel in traffic scene understanding by modeling complex relationships in graph-structured data. Applications include vehicle behavior classification across datasets [100], recognizing dynamic traffic police gestures [101], interpreting these gestures in real-time [102], understanding police intentions from visual cues [103], and recognizing actions of traffic participants in advanced driver-assistance systems [104].

The MR-GCN architecture for vehicle behavior classification [100] achieves sensor invariance and high accuracy: 99% on Apollo, 89% on KITTI, and 84% on Indian datasets. Combining spatial scene graphs and LSTM layers, it encodes spatial-temporal dynamics and outperforms baselines, demonstrating robustness across diverse datasets, even with fewer landmarks.

In [101], a gesture recognition method focuses on dynamic traffic police gestures using a spatial-temporal GCN (ST-GCN) with attention mechanisms and adaptive graph structures. It achieves 87.72% accuracy on the Chinese Traffic Police Gestures (CTPG) dataset, outperforming existing action-recognition methods.

Pose GCN [102] presents an online activity recognition method employing pose estimation and GCNs to interpret traffic police gestures in real-time frames. It achieves a response time of $716\mathrm{ms}$ and an accuracy rate of $97.52\mathrm{\%}$ on the TPGR dataset.

In [103], a system for recognizing traffic police intentions from visual cues achieves $87.72\mathrm{\%}$ OA on the TPGR dataset. The approach uses OpenPose [105] to extract key points, which are transformed into spatiotemporal maps processed by GCNs and modified transformers.

FIGURE 8. GAT-based license plate detection: An image of a car’s rear undergoes convolution to extract essential features, which are then refined by a GAT layer using an attention mechanism to determine the importance of neighboring features. The GAT operates on a graph representation, where each node is associated with a feature vector, and computes attention weights between nodes to aggregate information effectively. The saliency map is produced by fusing these attention-weighted features, guiding an RPN to accurately localize and identify the license plate. This sophisticated setup, combined with attention mechanisms to compute dynamic weights, enhances detection precision, ensuring reliable and accurate identification of license plates under various conditions. The integration of multiple attention heads helps capture different aspects of neighboring relationships, contributing to robustness in feature refinement.

The framework in [104] employs 3D human pose estimation and a dynamic adaptive GCN to recognize actions of traffic police, cyclists, and pedestrians. By optimizing object detection and pose estimation modules, it processes multiple objects simultaneously in real traffic scenarios, achieving $80\mathrm{\%}$ accuracy on the 3D-HPT dataset.

GCNs effectively model complex relationships in non-Euclidean data like graphs but face challenges with scalability due to high computational and memory demands on large graphs. They are prone to over-smoothing, where node features lose distinction after multiple layers, and require careful design to capture long-range dependencies, as standard architectures may not naturally handle distant node relationships.

2) GAT

GAT,introduced in [106], represents NN architectures for graph-structured data, incorporating attention mechanisms. Using masked self-attentional layers, GATs overcome the limitations of graph convolutions by allowing nodes to assign varying weights to neighbors' features. This avoids computationally expensive matrix operations like inversion and does not require a priori graph structure knowledge.

Figure 8 illustrates the GAT-based license plate detection process, where the attention mechanism refines features for accurate license plate identification in traffic scenes. For a graph $\mathcal{G}=\left(V,E\right),V$ represents the set of nodes,and $E$ represents the set of edges. Each node $v_i\in V$ is associated with a feature vector ${\mathcal{F}}_i\in {\mathbb{R}}^d$ ,where $d$ is the feature dimension.

The key idea behind GAT is to compute an attention weight for each neighbor of a node and use these weights to aggregate information from the neighbors. GAT employs a self-attention mechanism to calculate the attention weights for each neighboring node $v_j$ with respect to node $v_i$ , formulated as:

where ${\alpha }_{ij}$ is the attention score, $W$ is a learnable weight matrix, $\parallel$ denotes concatenation, $a$ is a shared learnable weight vector,and $f_{\text{LeakyReLU }}$ is a leaky ReLU activation function. Normalized attention weights are obtained using a softmax function:

where ${\mathcal{N}}_i$ is the set of neighbors of $v_i$ ,and ${\beta }_{ij}$ is the normalized attention weight for neighbor $v_j$ . GAT aggregates information from neighbors using the attention weights:

where $\sigma$ is an activation function (e.g.,ReLU),and ${\mathcal{F}}_i^{\prime }$ is the updated feature vector for node $v_i$ .

To capture diverse neighborhood relationships, GAT uses multiple attention heads. Outputs from all heads are concatenated and passed through a learnable weight matrix:

where ${\mathcal{F}}_i^{\prime \left(k\right)}$ is the output of the $k$ -th attention head, $\mathcal{K}$ is the number of heads, and || denotes concatenation.

GATs allocate attention within scene graphs, enabling precise object detection and tracking. They are applied to detect license plate numbers in urban environments [107], track pedestrians for traffic safety [108], enhance semantic segmentation, anomaly detection, traffic flow analysis, and improve classification accuracy in complex traffic scenes [109].

FIGURE 9. GIN procedure for traffic scene matching: The process begins with “Dataset Preprocessing,” where traffic datasets are converted into road scene graphs via a graph prediction module. This step involves cleaning, filtering, and transforming raw traffic data into a structured format suitable for graph construction. Concurrently, "Query Preprocessing" processes a traffic scene query through "Actor" and "Map Components" clusters, forming a scene graph of the specific road described in the input query. This involves identifying and classifying key elements of the traffic scene, such as vehicles, pedestrians, and road features. These preprocessed graphs are then used to construct the “Input Graph” (Graphx) and the “Isomorphic Matching Subgraph” (Graphy). Graph ${}_{\chi }$ represents the overall dataset graph, while Graphy is a subgraph extracted to match the query. The procedure culminates in the “Results,” where matched road scenes are displayed. This final step leverages the GIN to ensure precise matching and accurate representation of road scenes from diverse datasets, providing practitioners with reliable scene matches for analysis and decision-making.

APSEGAT [107] efficiently detects license plate numbers in crowded urban environments with diverse vehicles and complex scenes. It achieves a superior F-Score of $90\mathrm{\%}$ compared to YOLO’s 86% on the AMLPR dataset.

The GAM tracker [108] employs sparse candidate selection, graph attention maps, and distance matching loss for pedestrian tracking, achieving 94.99% MOTA on the Pets-mf dataset. It addresses pedestrian safety challenges and supports traffic statistics and abnormal behavior analysis in intelligent transportation systems.

SCENE [109] leverages heterogeneous GNNs and graph convolutions to encode diverse traffic scenarios, achieving 91.17 accuracy in binary node classification tasks on a custom large-scale dataset ("GAT_SCENE") with 22,400 sequences, each containing 3 seconds of temporal history. Performance and transferability are notably enhanced by incorporating edge features into the GAT operator.

GATs enhance GCNs by assigning varying importance to neighboring nodes via an attention mechanism for more nuanced relationship modeling. However, this mechanism is computationally expensive on large graphs, prone to overfitting when focusing on a few nodes, and struggles to capture long-range dependencies due to its local application.

3) GIN

GIN, introduced in [110], is a GNN designed to process graph-structured data by effectively capturing intricate graph topology. It enhances node representations by considering both a node's features and its neighbors' contributions, enabling precise structural differentiation and identifying subtle graph differences.

Figure 9 illustrates the GIN procedure for matching traffic scenes. Let ${\mathcal{F}}_v^{\left(0\right)}$ represent the initial feature vector of node $v$ . GIN begins by aggregating information from neighboring nodes and combining it with the node's own features. The aggregation employs an MLP, a fully connected neural network with multiple layers. The updated node representations are computed using the function $f_{\mathrm{MLP}}$ as follows:

where $\mathcal{N}\left(v\right)$ represents the neighbors of node $v,{ϵ}^{\left(l\right)}$ is a learnable parameter,and ${\mathcal{F}}_v^{\left(l\right)}$ is the representation of node $v$ at layer $l$ after being processed by the function $f_{\mathrm{MLP}}$ .

In some cases, a readout function aggregates node-level information to obtain graph-level embeddings. For example, graph-level embedding ${\mathcal{F}}_{\text{graph }}$ could be computed as the sum of all node embeddings:

where $L$ is the number of GIN layers.

GIN iteratively updates node representations using their own features and those of neighboring nodes. A learnable parameter ${ϵ}^{\left(l\right)}$ distinguishes between a node’s own features and its neighbors', enabling GIN to capture complex graph structures. This is particularly effective for graph classification tasks where topology is crucial.

GINs are widely applied in traffic scene understanding, including road scene-graph embedding [111], vehicle and pedestrian path prediction [112], traffic scene retrieval [113], automatic scenario detection [114], and real-time pedestrian path prediction [115].

Roadscene2vec [111] uses GCNs, GINs, and CNNs to enhance road scene-graph analysis for spatial modeling, graph learning, and risk assessment. For collision prediction, it achieves $88.12\mathrm{\%}$ (GCN), $80.28\mathrm{\%}$ (GIN),and 70.39% (ResNet-50) on 271-syn, and 90.95%, 78.03%, and 80.80%, respectively, on 1043-syn. For subjective risk assessment, evaluating perceived driver risk, it achieves 93.20% (GCN), 85.61% (GIN), and 69.38% (ResNet-50) on 271-syn, and $95.80\mathrm{\%},87.84\mathrm{\%}$ ,and $90.53\mathrm{\%}$ ,respectively,on 1043-syn.

Pishgu [112] introduces a lightweight network combining GINs with attention mechanisms for path prediction. It improves ADE/FDE by up to $42\mathrm{\%}/61\mathrm{\%}$ for vehicles (bird’s-eye view) and $23\mathrm{\%}/22\mathrm{\%}$ for pedestrians (high-angle view). Tested on the ActEV/VIRAT dataset, it achieves ADE and FDE scores of 14.11 and 27.96, making it a key resource for CyberPhysical Systems (CPS) applications.

RSG-Search [113] is a graph-based traffic scene retrieval system using sub-graph isomorphic searching for actor configurations and semantic relationships. It ensures dataset compatibility (e.g., nuScenes, NEDO), achieving full accuracy with low matching times (0-2 seconds). The RSG dataset includes 500 traffic scenes, 200,000 topological graphs, 6 node types (e.g., vehicle, pedestrian), and 25 relationship categories (e.g., 'passing-by', 'waiting-for').

The study in [114] presents expert-knowledge-aided representation learning for traffic scenarios using GIN and an automatic mining strategy. It enables effective clustering and novel scenario detection without manual labeling, achieving an AUC of $99.1\mathrm{\%}$ on a dataset simulated with OpenStreetMap.

CARPe [115] introduces a real-time pedestrian path prediction approach by combining GINs with an agile convolutional NN design. It achieves impressive results with an ADE of 0.80 and FDE of 1.48 on the ETH dataset, significantly improving speed and accuracy for applications such as autonomous vehicles and environmental monitoring.

GINs excel at distinguishing graph structures by capturing subtle differences between nodes and edges, making them effective for graph classification. However, they are prone to overfitting with limited data, requiring careful hyperpa-rameter tuning. GINs also face scalability and efficiency challenges on large or complex graphs due to their depth and computational demands.

G. CapsNet

CapsNet, introduced in [116], addresses CNN limitations by effectively handling spatial hierarchies between simple and complex objects. It encapsulates feature information (e.g., pose, texture, deformation) into neuron groups called "capsules," which use dynamic routing for enhanced feature representation and recognition. A capsule's activity vector represents the instantiation parameters of an entity (e.g., object or part), with its length indicating the probability of the entity's presence in the input.

To ensure the output vector of a capsule is a small-length vector if the probability of the entity being present is low and a long-length vector if it is high, a squashing function is used. It is typically defined as:

where ${\mathrm{s}}_j$ is the total input into the $j$ -th capsule,and $\parallel \cdot$ ${\left|\right|}_1$ denotes the L1 norm or length of a vector. This function ensures that the length of $v_j$ is between 0 and 1,thus representing a probability.

Dynamic routing allows a capsule to send its output to parent capsules in the next layer based on prediction agreement. Each lower-layer capsule predicts the output of higher-layer capsules using a transformation matrix $W_{ij}$ , expressed as ${\hat{\mathrm{u}}}_{j\mid i}=W_{ij}{\mathrm{u}}_i$ ,where ${\mathrm{u}}_i$ is the output of the $i$ - th lower-layer capsule,and ${\hat{\mathrm{u}}}_{j\mid i}$ is the prediction for the $j$ -th higher-layer capsule.

Capsules send outputs to parent capsules based on "routing by agreement," measured by the scalar product between the prediction vector and the parent's output vector. The coupling coefficients $c_{ij}$ are then updated based on the agreement and are defined by a softmax function over the initial logits $b_{ij}$ :

The total input ${\mathrm{s}}_j$ to the $j$ -th capsule is calculated as follows. It is a weighted sum of the predicted outputs:

Algorithm 1 shows the dynamic routing procedure to train the coupling coefficients $c_{ij}$ between each $i$ -th primary and $j$ -th output capsule [116].

FIGURE 10. Traffic scene classification with CapsNet: The input image is first processed through a ReLU convolution layer, followed by 32 primary capsules (’PrimaryCaps’). Capsules, fundamental units of CapsNets, encapsulate feature information like pose and texture, with the length of each capsule’s output vector representing the entity’s presence probability. These primary capsules are linked to 8 traffic capsules (‘TrafficCaps’) via weight matrices $W_{ii}$ , and dynamic routing is applied to selectively send outputs based on agreement. The length of each TrafficCaps vector, constrained by a squashing function to be between 0 and 1, influences the classification loss. $\left|\right|L_2\left|\right|$ normalization further enhances categorization, ensuring robust classification of traffic elements like police cars and pedestrians.

Algorithm 1 Routing Algorithm to Train the Coupling Coefficients Between Each Primary and Output Capsule

procedure $\mathrm{ROUTING}\left({\hat{u}}_j\mid i,r,l\right)$

for all capsule \( i \) in layer \( l \) and capsule \( j \) in layer \( \left( {l + 1}\right) \) :

对于层 \( l \) 中的所有胶囊 \( i \) 和层 \( \left( {l + 1}\right) \) 中的胶囊 \( j \) :

$b_{ij}\leftarrow 0$

for \( r \) iterations do

进行 \( r \) 次迭代

	for all capsule \( i \) in layer \( l : {c}_{i} \leftarrow  \operatorname{softmax}\left( {b}_{i}\right) \)

	对于层 \( l : {c}_{i} \leftarrow  \operatorname{softmax}\left( {b}_{i}\right) \) 中的所有胶囊 \( i \)

{softmax computes Equation 35}

	for all capsule \( j \) in layer \( \left( {l + 1}\right)  : {s}_{j} \leftarrow  \mathop{\sum }\limits_{i}\left( {{c}_{ij} \cdot  {\widehat{u}}_{j} \mid  i}\right) \)

	对于层 \( \left( {l + 1}\right)  : {s}_{j} \leftarrow  \mathop{\sum }\limits_{i}\left( {{c}_{ij} \cdot  {\widehat{u}}_{j} \mid  i}\right) \) 中的所有胶囊 \( j \)

	for all capsule \( j \) in layer \( \left( {l + 1}\right)  : {v}_{j} \leftarrow  \operatorname{squash}\left( {s}_{j}\right) \)

	对于层 \( \left( {l + 1}\right)  : {v}_{j} \leftarrow  \operatorname{squash}\left( {s}_{j}\right) \) 中的所有胶囊 \( j \)

{squash computes Equation 34}

	for all capsule \( i \) in layer \( l \) and capsule \( j \) in layer

	对于层 \( l \) 中的所有胶囊 \( i \) 和层中的胶囊 \( j \)

$\left(l+1\right):b_{ij}\leftarrow b_{ij}+{\hat{u}}_j\mid i\cdot v_j$

return $v_j$

These equations and the architecture help preserve detailed spatial information and enable the network to better understand the relationships and hierarchies between different parts of the objects.

Figure 10 illustrates the CapsNet procedure for traffic scene classification. The third layer, Traffic Capsules (Traf-ficCaps), contains 8 capsules, each as a 16-dimensional vector, fully connected to the previous layer's capsules. Dynamic routing ensures layer communication, while a squashing function bounds output vector lengths between 0 and 1 , indicating entity presence probabilities. Weight matrices $W_{ij}$ classify features into 8 traffic object classes.

CapsNets outperform traditional CNNs in robustness and generalization, excelling at capturing spatial relationships and complex interactions in traffic scenes, such as congested intersections. Applications include traffic sign detection [117], highway scene segmentation [118], and complex scenario recognition [119], offering deeper and more reliable insights into dynamic road environments.

TSDCaps [117] addresses CNN limitations for traffic sign detection,achieving $97.6\mathrm{\%}$ accuracy on the GTSRB dataset. It improves feature extraction, enhances reliability, and demonstrates resistance to adversarial attacks, making it well-suited for autonomous vehicle applications.

A scene segmentation model in [118], trained on Auckland Highway Images (AHI), achieves 74.61% accuracy. It enhances scene comprehension using matrix representations for pose and spatial relationships, reduces manual data manipulation, and addresses the challenging Picasso problem.

The authors of [119] proposed "ImprovedCaps," a two-step approach for complex scenes. It enhances traffic sign features through image processing before applying CapsNet for recognition, improving GTSRB accuracy by 2%- 5% in complex scenarios and achieving 96% overall accuracy.

In [120], "LiuCaps" is introduced for traffic-light sign recognition in autonomous vehicles. Trained on the TL_Dataset, it achieves 98.72% accuracy and a 99.27% F1-score, outperforming traditional CNNs while reducing training data needs and improving spatial relationship handling.

CapsNets excel at capturing spatial hierarchies and pose relationships, providing detailed object structure understanding compared to CNNs. However, they are computationally intensive, memory-demanding, and challenging to train, requiring complex optimization. Their limited scalability and efficiency on large datasets hinder adoption in resource-critical, large-scale applications.

H.HPO FOR DISCRIMINATIVE DL ARCHITECTURES

HPO optimizes model architecture and parameters like learning rate, network structure, and regularization, enhancing accuracy, generalization, and efficiency across datasets. Fine-tuning learning rates, anchor box dimensions (R-CNN, YOLO), and architectures (DETR, ViT) improves performance and pixel-level accuracy. In graph learning, tuning GCN layers and node features enhances spatial and structural relationship capture, improving predictions. Techniques like dropout, early stopping, and batch size optimization combat overfitting and computational constraints, while task-specific tuning strengthens multitask learning and temporal modeling for traffic scene understanding.

The learning rate controls weight updates: higher rates accelerate training but risk instability; lower rates improve precision but slow convergence. Batch size affects stability and memory: larger sizes enhance stability; smaller ones improve generalization. Epochs determine dataset passes: more epochs improve learning but risk overfitting, while fewer reduce training time but may underfit. Momentum accelerates training by smoothing updates, reducing oscillations, and avoiding local minima; higher momentum speeds convergence but risks overshooting, while lower momentum ensures stability. Weight decay (L2 regularization) prevents overfitting by penalizing large weights, promoting simpler models: higher values reduce overfitting but may underfit, while lower values allow flexibility but risk overfitting. Anchor scale adjusts predefined box sizes in object detection: larger scales improve the detection of big objects, while smaller scales enhance accuracy for small objects.

In the reviewed Fast R-CNN models, [22] used SGD with mini-batches of 2 images, 128 RoIs, a learning rate of 0.001 decayed after $30\mathrm{k}$ iterations,momentum of 0.9, and weight decay of 0.0005 . Reference [26] employed a 0.001 learning rate, batch size of 128 , and trained for 100 epochs.

For Faster R-CNN,[34] modified anchor scales $({80}^2,{112}^2$ , ${144}^2$ ) and aspect ratios(0.4,0.8,2.5),maintaining nine anchors per position. Reference [28] used a learning rate of 0.001 , batch size of 32, 100 epochs, momentum of 0.9 , and weight decay of 0.0005 .

In Mask R-CNN, [51] set the learning rate to 0.02 , weight decay to 0.0001, momentum to 0.9, batch size of 16, and trained for $900\mathrm{k}$ iterations. Reference [57] used a learning rate of 0.001 , batch size of 1 , and trained for 50 epochs.

For YOLO models, [121] used a learning rate of 0.001 decayed by 0.1 per epoch, momentum of 0.9 , weight decay of 0.0005, 8000 max_batches, and batch size of 32 . Reference [122] used a learning rate of 0.01 , final rate of 0.2 , momentum of 0.937, weight decay of 0.0005, 110 epochs, and batch size of 12. Reference [123] utilized a learning rate of 0.0002,Adam optimizer with $\beta 1=0.5,\beta 2=0.999$ ,and batch size of 32 .

Among ViT models,[89] used $16\times 16$ patches, 768 embedding dimension, 12 layers, 12 attention heads, Adam optimizer with a 0.001 learning rate, and 100 epochs. Reference [91] used $32\times 32$ patches,12 attention heads, 768 embedding dimension, and Rectified Adam, training for up to 1000 epochs with early stopping. Reference [92] used $16\times 16$ patches, 6 layers, 8 attention heads, and trained with a 0.0002 learning rate over 80 epochs.

For DETR models, [94] used ResNet-50/101 backbones, a transformer with 6 encoder/decoder layers, 256 hidden dimensions,8 attention heads,AdamW optimizer with ${10}^{-4}$ learning rate, batch size of 16 , and 300 epochs. Reference [95] used ResNet-50, AdamW optimizer, 0.0001 learning rate, batch size of 4,300 query positions, and data augmentation. Reference [96] used ResNet-50,AdamW, ${10}^{-4}$ learning rate, batch size of 8, adjusted rates on plateau, and data augmentation. Reference [97] used ResNet-50, AdamW, ${10}^{-4}$ learning rate,batch size of 32,and 50 epochs with scheduled rate reductions.

I. COMPARISON OF DISCRIMINATIVE DL ARCHITECTURES

Here's a shorter version that retains all the key information:

Table 2 compares various discriminative DL architectures. In classification, models are evaluated by overall accuracy for traffic scene recognition tasks. The Receptive Field NN [6], an early DL application for road sign classification, achieved a modest $47.7\mathrm{\%}$ accuracy on a custom dataset ("RFNN_TSR"), illustrating early limitations. More recent CNN-based methods, like 2LConvNet ms 108-108 [7], and CapsNet methods, such as LiuCaps [120], reached 97.83% and $98.72\mathrm{\%}$ accuracy,respectively,on another traffic sign dataset.

For object detection, Table 2 highlights performance improvements of Fast R-CNN over standard R-CNN, with AllLightRCNN [32] achieving 94.20% mean accuracy ("All-LightRCNN_DS"), a 16.4% gain over R-CNN's 65.76% mAP ("RCNNs_Detection"). Mask R-CNN and Faster R-CNN achieved mAPs of 74.30% and 76.30% [57]. Faster R-CNN models on COCO 2017 achieved AP scores from 40.2% to 44.0%, with Faster R-CNN-FPN-R101+ (108 epochs) [97] obtaining the highest AP of $44.0\mathrm{\%}$ . The YOLO family (YOLOv1 to YOLOv8) showed strong results, with YOLOv7 [73] achieving 98.77% AP ("ShokriCollec-tion_DS") for real-time vehicle detection. YOLOv3 [84] performed well on the PSU dataset for aerial traffic object detection,achieving $96.5\mathrm{\%}$ AP,outperforming Faster R-CNN’s 73.9%. DETR models also excelled on COCO- 2017, with SMCA-R50 (108 epochs) [97] achieving an AP of 45.6%, surpassing Faster R-CNN by 1.6%. Results from YOLO and DETR demonstrate their state-of-the-art performance, unifying bounding box prediction and classification in a single step.